1. Field of the Invention
The present invention relates to an apparatus for supporting evaluation of usability related to an operation with respect to a device or instrument which receives a user input and presents at least visual information in accordance with a content of the operation. An example of such an instrument would be a device that includes a graphical user interface (GUI).
2. Description of the Related Art
Instruments which present visual information related to user operation in order to supply various information, such as an instrument state and predetermined calculation result, to a user have become common and widespread. Examples of such instruments commonly found in most offices include copying machines, computers, and air conditioners. For example, most modern copying machines include a panel for setting the number of copies to be made and various copying functions (double-sided copying and reduction/enlargement), and a user operates the instrument based on the supplied visual information. Here, the visual information presented is not limited to dynamic information displayed on a liquid crystal display (LCD) such as information indicating the number of copies requested, but includes printed and fixed information, such as, for example, numerals xe2x80x9c1xe2x80x9d, xe2x80x9c2xe2x80x9d, . . . printed on a ten key keypad disposed for setting the number of copies to be made in the copying machine.
As instrument functionality has grown more sophisticated, operations have increasingly become complicated. International efforts have been made to facilitate improvement of instruments by providing indexes for evaluating the usability of instruments, such as user-friendliness of using the instrument, effectiveness of use of the instrument, efficiency, and satisfaction given to a user of the instrument. For example, the ISO 9241-11 standard (Ergonomic requirements for office work with visual display terminals (VDTs)-Part 11: Guidance on usability) defines evaluation measures related to usability from the viewpoints of effectiveness, efficiency, and satisfaction, and device usability can be enhanced through such evaluation. Here, usability indicates the degree of effectiveness, efficiency, and user satisfaction in achieving a designated object under designated circumstances of a certain product by a designated user. The effectiveness indicates accuracy and completeness in achieving a designated result, efficiency measures the amount of resources consumed in association with the accuracy and completeness in achieving the target by the user, and satisfaction measures comfort and acceptability for a user who uses a computer system and the computer system, that is, the degree to which a user is free from frustration and is able to maintain a positive attitude toward product use.
Known methods of performing evaluation of a task using information from users who have completed the relevant task include questionnaires in which users independently read and answer questions and interviews where users are asked questions and provide answers to evaluators. With a questionnaire, although a specific event at a task achieving time or an entire impression at a task end time can be better understand, a relationship between the operation content and problem is often unclear, and it is often difficult to use the information to improve the device.
There are known apparatuses for recording a user operation log and diagrammatically displaying the log record to facilitate analysis (e.g., Japanese Patent Application Laid-Open No. 8-153022). Moreover, there are techniques for recording and reproducing the operation log between the system, which is an evaluation object, and the user and listing and displaying the state log of the system (e.g., Japanese Patent Application Laid-Open No. 2001-51876).
Furthermore, there is another related art system in which user testing is employed. In user testing, a test subject regarded as a user actually operates the instrument, a test subject which attempting to perform a task is observed by an observer, the observer presumes what the test subject is going to do, and the observer notes steps at which the usability drops.
For example, when it is observed that the test subject""s eyeshot moves over an operation panel, it is presumed that the test subject is searching the panel for information or a means to input a desired command. When this takes an excessively long amount of time, it is presumed that the operation of the panel is not clear and the panel design is not efficient for operation, and therefore the usability of such a panel is low.
However, in these related-art methods of evaluating usability, an evaluator must refer to records of the operation log, state log of the system, and test subject""s state to conduct the evaluation. Therefore, evaluator""s arbitrary judgment is easily included, and the final results tend to depend on the individual knowledge of each evaluator.
When there are a plurality of evaluators, particularly when there are many evaluation objects, uniform evaluation is not easily obtained, and the criteria or components to be improved are not easily distinguished. Moreover, to obtain the questionnaire or interview information from the test subject, only the content that the test subject is aware of can be extracted. Therefore, there is a large possibility that a test subject will unconsciously edit their thoughts or comments in order to describe the state after the end of the operation. Furthermore, with the known usability evaluation methods it is not always possible to obtain indexes of the usability such as effectiveness, efficiency, and satisfaction, which conform with the ISO 9241-11 standard.
Additionally, even if such indexes as effectiveness, efficiency, and satisfaction are obtained, because the content or definition of the index differs for different types of business, it is difficult to standardize evaluation. Moreover, effectiveness, efficiency, and satisfaction can be defined in each situation of use of the evaluation object. Therefore, when effectiveness, efficiency, and satisfaction data are not obtained according to the same definitions among the objects to be compared/evaluated, the meaning of quantification and evaluation is weakened. Additionally, there has been a demand for an ability to evaluate individual components of a task to evaluate their negative effect on the overall usability so that the specific components can be improved. Therefore, the burden on evaluators is heavy, and extensive training of evaluators is necessary.
The present invention was developed in consideration of the above-described circumstances, and provides an apparatus for supporting evaluation of usability so that usability can be easily and efficiently evaluated in consideration of circumstances regarding individual businesses in a mode suitable for evaluation conforming to ISO standards.
According to one aspect of the present invention, there is provided an apparatus for supporting evaluation of usability in achieving a given task with respect to an instrument including a display unit for displaying information, the apparatus comprising means for acquiring information of an operation log which is related to an operation performed by a user and which includes at least one item of operation log data collected during performance of the task; and index evaluation means for generating at least one effectiveness index related to accuracy and completeness in achieving a target of the task, efficiency index related to a resource consumed in association with the accuracy and completeness in achieving the target of the task, and satisfaction index related to satisfaction of a task achiever in achieving the target of the task. The index evaluation means evaluates and generates the effectiveness, efficiency, and satisfaction indexes using a statistical calculation involving at least one data included in at least one of a plurality of measurement items including the operation log information.
Moreover, the apparatus may further comprise means for acquiring at least one item of user biometric data (data related to the user as a living being), and the index evaluation means may also preferably evaluate the effectiveness, efficiency, and satisfaction indexes using a statistical calculation involving at least one item of data included in at least one of a plurality of measurement items including the information of the operation log and the information on the living organism. Furthermore, the apparatus further comprises means for acquiring satisfaction evaluation of the user concerning the achieving of the task, and the index evaluation means also preferably evaluates the effectiveness, efficiency, and satisfaction indexes of each subtask using at least one data item included in a plurality of measurement items related to the satisfaction evaluation. Here, the means for acquiring the satisfaction evaluation may be means for inputting a result of questionnaire or interview, or means for inputting biometric data of the user.
Furthermore, the apparatus preferably comprises means for performing a predetermined statistical calculation in order to generate each effectiveness evaluation result, efficiency evaluation result, and satisfaction evaluation result based on at least one effectiveness index, efficiency index, and satisfaction index generated by the index evaluation means.
Moreover, in the aspect of the present invention, the content of the statistical calculation in evaluating the effectiveness, efficiency, and satisfaction indexes in the index evaluation means can be changed. Furthermore, the content of the predetermined statistical calculation for generating the effectiveness, efficiency, and satisfaction evaluation results can be changed. Ability to change, alter, or customize the content of the statistical calculation can be changed in this manner enables the evaluation to be set, modified, or customized according to the specific circumstances of a particular business.
According to another aspect of the present invention, there is provided a method for supporting evaluation of usability in achieving a given task with respect to an instrument including a display unit for displaying information, the method comprising a step of acquiring information of an operation log which is related to an operation performed by a user and which includes at least one data of the operation log during the achieving of the task; and an index evaluation step of generating an effectiveness index related to accuracy and completeness in achieving a target of the task, an efficiency index related to a resource consumed in association with the accuracy and completeness in achieving the target of the task, and a satisfaction index related to satisfaction of a task achiever in achieving the target of the task. The index evaluation step comprises a steps of evaluating and generating the effectiveness, efficiency, and satisfaction indexes using a statistical calculation performed on at least one data item included in at least one of a plurality of measured items, including the operation log information.
Here, the method further comprises a step of generating an effectiveness evaluation result, an efficiency evaluation result, and/or a satisfaction evaluation result based on at least one effectiveness index, efficiency index, and satisfaction index generated by the index evaluation step, using a predetermined statistical calculation.
According to further aspect of the present invention, there is provided a program which supports evaluation of usability in achieving a given task with respect to an instrument including a display unit for displaying information and which enables a computer to perform a procedure of acquiring information of an operation log related to an operation performed by a user and including at least one data of the operation log during the achieving of the task; and an index evaluation procedure of generating at least one effectiveness index related to accuracy and completeness in achieving a target of the task, efficiency index related to a resource consumed in association with the accuracy and completeness in achieving the target of the task, and satisfaction index related to satisfaction of a task achiever in achieving the target of the task. The index evaluation procedure comprises a step of performing a statistical calculation evaluating and generating the effectiveness, efficiency, and satisfaction indexes using at least one data item included in at least one of a plurality of measurement items including the operation log information.
Here, the computer is preferably caused to further execute a procedure of generating effectiveness evaluation results, efficiency evaluation results, and satisfaction evaluation results based on at least one effectiveness index, efficiency index, and satisfaction index generated by the index evaluation procedure using a predetermined statistical calculation.